Nominal

Binary

Ordinal

\[\downarrow interval-scaled\]

\[\begin{matrix}&Class\_ Rank&Rich\_Rank\\sample_1&1&1\\sample_2&4&2\\sample_3&3&3\\sample_4&2&3\end{matrix}\rightarrow\begin{matrix}Class\_ Rank&Rich\_Rank\\\frac{1-1}{4-1}=0&\frac{1-1}{3-1}=0\\\frac{4-1}{4-1}=1&\frac{2-1}{3-1}=\frac{1}{2}\\\frac{3-1}{4-1}=\frac{2}{3}&\frac{3-1}{3-1}=1\\\frac{2-1}{4-1}=\frac{1}{3}&\frac{3-1}{3-1}=1\end{matrix}\]

Interval

Ratio

Mixed

Mixed_DataType

K-means clustering

1. Given \(k\) initial points randomly.

2. Calculate distance between all samples and the \(k\) points.

3. Distribute every sample to its closest point of the \(k\) points, thus forming \(k\) clusters.

4. Updating the \(k\) points by calculating central positions of \(k\) clusters.

5. Repeat step 2 to step 4 until the \(k\) points don’t change any more.

import numpy as np
from sklearn.cluster import KMeans

# Generate ndarray of shape (100, 3)
X = np.random.uniform(0, 10, size=(100, 3))
X = np.round(X, decimals=2)

# initiate k=3
kmeans = KMeans(n_clusters=3)
kmeans.fit(X)

labels = kmeans.labels_  # cluters for every sample
centers = kmeans.cluster_centers_  # center coordinates of every cluster

for i in range(len(X)):
    print("sample {} belongs to {}".format(X[i], labels[i]))

for i in range(len(centers)):
    print("center coordinate of cluster {} is {}".format(i, centers[i]))

Hierarchy clustering

The 5 distance definitions of clusters in hierarchy clustering is shown on the pictures below.

import numpy as np
from scipy.cluster import hierarchy
import matplotlib.pyplot as plt


# Calculate a distance matrix. This matrix should be either upper triangular or lower triangular matrix with out diagonal values.
half_distance_matrix = np.array([[0, 0, 0, 0, 0],
                                [2.33, 0, 0, 0, 0],
                                [3.15, 1.30, 0, 0, 0],
                                [1.90, 1.50, 3.70, 0, 0],
                                [3.01, 0.47, 1.40, 1.82, 0]]).T

# np.tril_indices(half_distance_matrix.shape[0], -1) returns values' indices of lower triangular matrix without the diagonal values
# np.triu_indices(half_distance_matrix.shape[0], 1) returns values' indices of upper triangular matrix without the diagonal values
# hierarchy.linkage() only receives a compressed 1-dimension array.
compressed_distance_matrix = half_distance_matrix[np.triu_indices(half_distance_matrix.shape[0], 1)]
linkage_matrix = hierarchy.linkage(compressed_distance_matrix, method='single')

plt.close('all')
def link_color_func(*args):
    return 'cornflowerblue'
dendrogram = hierarchy.dendrogram(linkage_matrix, labels=['1', '2', '3', '4', '5'], link_color_func=link_color_func)
plt.xlabel('Samples Indices')
plt.ylabel('Distance')
plt.title('Hierarchical Clustering Tree')
plt.show()

The argument method can be “single”, “complete”, “average”, “centroid”, “ward” with respect to the figures above.

The calculation for similarity matrix would be computationally expensive.

Density-based method

Eps: Maximum radius of the neighborhood.

MinPts: Minimum number of points in an Eps-neighborhood of that points.

Core point: Point with enough density.

Border point: Point without enough density but is neighbor of core point.

Directly density-reachable: Point \(p\) is directly density-reachable from point \(q\) if and only if \(q\) is a core point and \(p\) is a neighbor of point \(q\).

Density-reachable: Point \(p\) is directly density-reachable form point \(q\), and point \(o\) is directly density-reachable from point \(p\). Then \(o\) is density-reachable from \(q\).

Algorithm (DBSCAN):

1. Arbitrary select a point \(p\).

2. Retrieve all points density-reachable from \(p\).

3. If \(p\) is a core point, a cluster is found. If not, visit the next point of database.

4. Continue the process until all of the points have been processed.

Fuzzy clustering

1. Randomly choose \(k\) samples to initialize \(k\) clusters.

2. Calculate probability of each sample belongs different clusters by the formula below and thus forming partition matrix.

\[p_{ij}=\frac{\frac{1}{dist^2(i,j)}}{\sum_{j\in J}{\frac{1}{dist^2(i,j)}}}\], where \(i\) represents sample \(i\), and \(j\) represents cluster \(j\).

partition matrix:

3. Updating cluster by the formula below.

\[x_j=\frac{\sum_{i\in I}{x_ip_{ij}^2}}{\sum_{i\in I}{p_{ij}^2}}\], where \(x_j\) represents the x-axis coordinate of cluster \(j\). This formular should be applied to every axis of a cluster.

4. Repeat step 2 and step 3 until the partition matrix converges.

Compared to K-means, fuzzy clustering has advantages:

1. Less sensitive to outliers. The centroid updating is influenced by probability instead of distance.

2. Less sensitive to initial clusters.

But the shortcoming is that fuzzy clustering is computationally expensive as calculating a sample’s probability to every clusters.

import numpy as np
import skfuzzy as fuzz

data = np.random.uniform(0, 10, size=(100, 4))

n_clusters = 3
fuzz_cmeans = fuzz.cluster.cmeans(data.T, c=n_clusters, m=2, error=0.005, maxiter=1000)  # fuzz_cmeans is a tuple including 7 items.
fuzzy_centroids = fuzz_cmeans[0]
fuzzy_probs = fuzz_cmeans[1]  # partition matrix with shape (3, 100)

print(fuzzy_centroids)  # coordinates of n_clusters clusters
print(fuzzy_probs.argmax(axis=0))  # each sample's cluster

m in fuzz.cluster.cmeans() is another parameter which is set to 2 in order to keep consistent with the formulas above.